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Abstract — In this paper, we give a distributed joint source 
channel coding scheme for arbitrary correlated sources for 
arbitrary point in the Slepian-Wolf rate region, and arbitrary 
link capacities using LDPC codes. We consider the Slepian-Wolf 
setting of two sources and one destination, with one of the sources 
derived from the other source by some correlation model known 
at the decoder. Distributed encoding and separate decoding is 
used for the two sources. We also give a distributed source coding 
scheme when the source correlation has memory to achieve any 
point in the Slepian-Wolf rate achievable region. In this setting, 
we perform separate encoding but joint decoding. 

Index Terms — Distributed Source coding, joint source-channel 
coding, LDPC codes, Slepian-Wolf. 

I. Introduction 

DISTRIBUTED source coding (DSC) refers to the com- 
pression of the outputs of two or more physically sep- 
arated sources that do not communicate with each other 
(hence the term distributed coding). These sources send their 
compressed outputs to a central point (e.g., the base station) 
for joint decoding. Driven by a host of emerging applications 
(e.g., sensor networks and wireless video), DSC has recently 
become a very active research area - more than 30 years after 
Slepian and Wolf laid its theoretical foundation [1]. 

Wyner first realized the close connection of DSC to channel 
coding and suggested the use of linear channel codes as a 
constructive approach for Slepian-Wolf coding in his 1974 
paper [25]. Wyner's scheme was only recently used in [2] 
for practical Slepian-Wolf code designs based on conventional 
channel codes like block and trellis codes. If the correlation 
between the two sources can be modelled by a binary channel, 
Wyner's syndrome concept can be extended to all binary linear 
codes; and state-of-the-art near-capacity channel codes such 
as turbo [26] and LDPC codes [23] [24] can be employed to 
approach the Slepian-Wolf limit. 

Slepian-Wolf rate region is the rate region in which there 
exist encoding and decoding schemes such that reliable trans- 
mission can take place, which was given in [1] as i?i > 
H(X\Y) , R 2 > H(Y\X), and R 1 + R 2 > H(X,Y), where 
Ri and R 2 are the source encoding rates of the two sources X 
and Y, and H(X\Y) and H(Y\X) the conditional entropies of 
the sources, and H(X, Y) the joint entropy of the two sources. 
The most important points in the Slepian-Wolf rate region are 



the asymmetric point ( in which one source, say X, is encoded 
at the rate that equals its unconditional entropy (H(X) ) and 
the other source, say Y, is encoded at the rate equal to its 
conditional entropy (H(Y\X) )), and the symmetric point (in 
which both sources are encoded at the same rate H(X, Y)/2). 
Constructive approaches (e.g., [5]) have been proposed for 
the symmetric point of the boundary of the Slepian-Wolf 
rate region, for the asymmetric point of the boundary of the 
Slepian-Wolf rate region (example [2] [7] [12] [18]) and also 
for any point between on the boundary (for example [4] [13] 
). Most of the cases in the literature deal with correlation 
model being a binary symmetric channel, though some work 
on arbitrary correlation has also been done (for example [4]). 

After the practical scheme of [2], there was a lot of work 
in the direction of the practical code construction due to 
its application in sensor networks [5]. Some of the works 
relating to this field are [4][6][7][8][9][10][11][12]. In [4], 
practical construction using syndrome approach was given for 
arbitrarily correlated sources. There is also another scheme not 
using the syndrome approach, but rather encoding in the same 
way as the channel coding, by multiplying at the encoder by 
the generator matrix as is given in [13]( The idea of using 
such approach is also there in [3]). They consider in the 
paper source correlation model as BSC with error probability 
p. To encode, a non-uniform systematic LDPC code with 
generator matrix of size n x n(l+h(p)) is used (where h(p) 
is the binary entropy of p), and at the encoder of the first 
source, say X\, 1/a fraction of the information bits, and (1- 
1/a) fraction of the parity bits are sent. At the encoder of 
the other source, say X 2 , the remaining (1-1/a) fraction of 
information bits and the remaining 1/a fraction of the parity 
bits are sent. Decoding is performed as follows: At decoder 
of X\, we have 1/a fraction of information bits of X with 
infinite likelihood, (1-1/a) fraction of parity bits with infinite 
likelihood, (1-1/a) fraction of information bits with ln((l-p)/p) 
likelihood, from which we decode X\, and similarly we can 
decode X 2 . Note that the remaining parity bits from X 2 are 
not used in the decoding of X\, they are considered as being 
punctured. They also suggest to use unequal error protection 
for LDPC since the information bits are more important. 
More important bits have higher degrees while the rest have 
lower degree. Hence, they design systematic LDPC code with 
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distinct variable node degree distributions for information and 
parity bits. Our scheme will extend upon this basic idea of 
doing source coding in the same way as traditional channel 
coding scheme. 

There has also been some work in the direction of joint 
source channel decoding of correlated sources over noisy 
channels (for example [6], [19], [22], [20], and [21]) but none 
of the work till now covers the problem in that generality using 
LDPC codes. It has been proven in [11] that LDPC codes can 
approach the Slepian Wolf bound for general binary sources, 
and hence the choice of LDPC codes [23] [24] in this work. 

Most of the cases in the literature deal with correlation 
model being a binary symmetric channel, though some work 
on arbitrary correlation has also been done (for example [4]) 
and also for correlation having Markov structure (for example 
[16] [18]). To the best of our knowledge, no work for arbitrary 
point in the Slepian-Wolf rate region is there in the literature 
although there are attempts for asymmetric point in the rate 
region( for example [16] [18]). 

The paper is organized as follows: We will start with 
giving the scheme of joint source channel coding for the 
source correlation model being being arbitrary memoryless 
with arbitrary link capacities at arbitrary point in the Slepian- 
Wolf Rate region in Section [II] Then, we give some simulation 
results for this scheme in Section III Following this, we give 
a scheme for doing distributed source coding at arbitrary rates 
in the Slepian-Wolf Rate Achievable region when the source 
correlation has memory in Section [IV] and some simulations 
for the scheme are given in Section TV) Finally, we give some 



concluding remarks in Section VI 



II. Joint Source Channel Coding Scheme for 
Arbitrary Source Correlation 

Consider first that we encode Xi as follows: X\ is fed into 
systematic LDPC code of Rate R\. The encoder sends the 
corresponding parity bits and a 1/a fraction of the information 
bits. These parity bits involve the bits needed for source as 
well as channel. Let Pi be the parity bits needed for source 
and P 3 the parity bits needed for the channel for source X\. 
The compression rate of source X\ is Rx t = k l a + Pl , and the 

where k 



corresponding LDPC code has rate R\ = k+Pi+P 
is the number of bits of X\ and of X^ at the input. The same 
procedure with a few modifications is applied at the source X^. 
For this source, we use a systematic LDPC code of rate i?2- 
The encoder sends the related parity bits and the remaining 
1-1/a fraction of the information bits. Similarly, assume that 
the parity bits are P2 for the source coding and P4 for the 
channel coding at the source X%, we have the compression 
rate and the LDPC code rate equal to Rx 2 = (a ~^ /a+P2 

and i?2 = k+p 2 +Pi res P ec ti verv - 

The procedure as we discussed above involves two LDPC 
codes. However, we are interested in designing a single 
channel code for both sources. So, we will make a single 
LDPC matrix having the information bits, and all parity bits 
corresponding to Pi, P%, P3, and P4. 

Lemma 2.1: Pi + P 2 = \t(Rx l + Rx 2 "1 ) 

Proof: Add Rx 1 and Rx 2 given above to get the required 



Lemma 2.2: P 3 + P 4 = k[R Xl {^- -1)+Rx 2 {^- 
Proof: 
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Let the rate of the LDPC code needed is R. With this parity 
check matrix, we will generate required parity bits for each 
source. Now, as we can see, the modified algorithm is to 
encode both the sources with a single LDPC code, and at 
encoder of X\, send the first fraction 1/a of the information 
bits, and the first 1/b fraction of the parity bits, and at encoder 
of X2, send the remaining 1-1/a information bits and last 1/c 
fraction of the parity bits (Figure [TJ. Also keep in mind that to 
satisfy the Slepian-Wolf Compression bounds, R Xl and Rx 2 
(the source compression rates of the two sources) have to be 
in the Slepian-Wolf rate region. At the decoder of the each 
source, we use the information bits coming from both the 
sources, and the parity bits from that source, and the remaining 
bits are considered as punctured. The parameters a,b,c,R, R Cl , 
R C2 (the channel rates of the two channels), Rx 1 and Rx 2 are 
related as in ([T}. Cf and Cb are the capacities of the channel 
from the source Xi to X2 and from source X2 to source Xi 
respectively. 
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As we can see that punctured bits can be further decreased 
by taking the total parity bits as the maximum number of parity 
bits being used and which equals maximum of Pi + P3 and 
Pi + Pi- Hence we perform the second step according to 
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R <- R' 

The encoding and decoding functions are explained in detail 
below: 

Encoding: 

The scheme is explained in the figure [T] At the encoder of 
X\, we send the first 1/a fraction of information bits and first 
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Fig. 1. Parallel Channel for (a) source X\ and (b) source X2 



1/b fraction of the parity bits. At the encoder of source X%, 
we send the remaining 1-1/a fraction of information bits, and 
last 1/c fraction of parity bits. The LDPC matrix made is of 
rate R . We have four variables here a,b,c,R for given Rx x , 
Rx 2 , Rci an d Rc 2 f° r which we use the set of equations ([TJ 
to solve them explicitly. It can also be seen that 



fc + fc(A-l)(i) 
k 



Ri = 



1 



Fig. 2. (a) Equivalent Model of Figure |TJb), and (b) Special case of this 
model as c=l 



Fig. 3. (a) Equivalent Model of Figure Qa), and (b) Special case of this 
model as b=l 



Consider channels of Figure [JJb). In Figure |2ja), we have 
an equivalent model with ri2=n-(l-l/c)(n-k) , where k=nR, 
and the parity bits are P2 and P4. According to [14], the 
performance of decoder do not change after puncturing if R/C 
do not change. It is easy to see using ([TJ that the model of 
source X2 is similar to test model in Figure |2jb) when c = 1. 
We will show that the decoder performance of the model of 
Figure |TJb) and figure |2|b) are the same in the next few 
lemmas. Similarly, we will show that the decoder performance 
of the model of Figure [TJa) and figure [3jb) (its equivalent 
model when b=l) which means that decoder performance do 
not change with the choices of a,b,c. 



Lemma 2.3: The ratio of rate and capacity are same for 
Figure |2ja) (or Figure [T|b)) and Figure |2|b) are the same 
Proof: As 



C-x = (1 — R 2 )R C2 
R R 



-R2-Rc 2 (l 
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CtR, 



f-n.ci 



R+(1-R)(l) 
C = (1 — R)R C2 + RCfR Cl 

After some manipulations, we get ^ = || using Equations 
0. ■ 

Lemma 2.4: The ratio of rate and capacity are same for 
Figure [TJa) and Figure [3jb) are the same. 
Proof: As 



1 R\ 
C\ = (1 — Ri)R Cl + i?i(l )CbR C2 + R C i — 

R = - 
1 R+(1-R)(l) 

C = (1 — R)R Cl + RCbR C2 
After some manipulations, we get = ^ using Equations 



It is also easy to see that the bits sent satisfy equation ([TJ, 
the total bits that are sent through encoder X\ are 

Rxi 



a R b 



Rr 



and the total bits that are sent through encoder X2 are 

fc(l 



a R c R Co 



Hence all the equations in equation ([TJ makes good sense. As 
we can see that punctured bits can be further decreased by 
taking the total parity bits as the maximum number of parity 
bits being used and which equals maximum of Pi + P3 and P2 
+ P4. Hence we perform the second step according to equation 
Q. 

We still need to find C/ and Cb for solving the above 
equations. Consider the forward model as in figure [4] The 
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Source X„ 



Fig. 4. Forward Model 

capacity of the forward model is given by 

Cf = h( fe(p 1 l) _ h(p2) ) - flhfa) - (1 - P)h( P 2) 

1 + 2 1 -P1-P2 
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Mpi 1 )-Mp2) P2 

where = 1+2 l^ 1 ^ 2 

^ 1-P1-P2 

If the probability that (X\ — 0) is a, then the backward 
channel is given as in figure [5] 




Fig. 5. Backward Model 



III. Simulations of Joint source channel coding 

FOR ARBITRARY MEMORYLESS CORRELATION 

As a first example, take p\ = .05, p 2 — .185 and a =.4 
in Figure |4] This gives a joint entropy of two sources as 1.5. 
Solving C f and C b gives C f = 0.4998, C b = 0.4839. Also take 
the source rates Rx x = -8, Rx 2 = -7, and the channel rates 
as R Cl = .9 and R C2 = .94. Putting all these in equations ([T| 
and |2]), we get the parameters a,b, c and R as 1.58, 1.4754, 1 
and .7259 respectively. These parameters should ideally work 
for this scheme. To simulate, we use p% = .05 and a =.4 in 
Figure |4] but we keep p 2 as a variable, changing which we 
vary the joint entropy of the sources. Also, choose the both 
the channels as BSC with error probabilities .0129 and .0069 
respectively so that both the channels rates are the same as 
the capacity. The simulation result is shown in Figure [6] 
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Hence, the capacity of the backward Channel is given by 
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fc(pj) — hip's) 
1 + 2 1 -P'l-P2 



It is clear that I have given separate parameters for source 
and channel encoding so that they can be separately decided, 
and then use equations ([TJ, and then equations |2]) to get all 
the relevant parameters of the code and design. 

According to [14], for any rates Ri and R2 that < R\ < 
i?2 < 1, there exists an ensemble of LDPC codes with the 
following property: The ensemble can be punctured from rate 
Ri to R 2 resulting in asymptotically good codes for all rates 
Ri < R < R 2 - Hence, we design punctured codes according 
to [14] for rate min(i?i,i?2) and that will work for both and 
this is the rate we get after the step 2 using the equations Q. 

Decoding: 

The decoder needs to determine Xi from information bits of 
Xi (partly that were sent from encoder of Xi and partly sent 
from encoder of X 2 ), and the parity bits Pi and P3. Likelihood 
of these parity bits are decided by the channel noise alone, and 
similar for the information bits coming from the encoder of 
X%, while the information bits from the encoder of X 2 will 
have the effect of the BSC in the path from X\ to X 2 also. 
Similar procedure will decode the source X 2 . 
Remark: 

The code construction(irregular LDPC codes) and decoding 
has to be done in the way suggested in [15] which gives a 
scheme using density evolution with erasures and errors which 
is the model of our scheme. So, this model gives the density 
evolution analysis for this case, and we can choose the degree 
distribution and decoding parameters according to this density 
evolution analysis. 
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Fig. 6. Simulation Results 
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Fig. 7. Simulation Results 

As a second example, consider the model for p\ = .09, 
p 2 = .143 and a = A. This gives a joint entropy of two sources 
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as 1.5. Solving C f and C b gives Cf = 0.4839, C b = 0.4703. 
Also take the source rates Rx 1 = -8, Rx 2 = -7, and the channel 
rates as R Cl = .85 and R C2 = .9. Putting all these in equations 
([TJ and Q, we get the parameters a,b, c and R as 1.5392, 
1.4665, 1 and .7005 respectively. To simulate, we use pi = .09 
and a =.4 in Figure [4] but we keep p2 as a variable, changing 
which we vary the joint entropy of the sources. Also, choose 
the both the channels as BSC with error probabilities .0215 
and .0129 respectively so that both the channels rates are the 
same as the capacity. The simulation result is shown in Figure 

IV. Distributed Source Coding Scheme for 
Markov Source Correlation 

The correlation between the sources has a memory defined 
by a Markov model. The two source sequences be Xi and 
Xi as before, with X2 = X\ + N, where N is generated by 
a Markov model A = (^4, B, tt) , and addition is modulo two 
addition. The model is characterized by a set of states Sj, < 
j < S — 1, the matrix of transition probabilities among states 
[A=(a,ij), with dij = Pt(Sj\Si) the probability of transition 
from state Si to Sj], and the list giving the bit probability to 
associate with each state [B={bj V ), with bj V — Po(v\Sj) the 
probability of getting output v in state Sj]. tt is the initial 
distribution of each state. This model has been taken for the 
source compression in [16] but this solves the asymmetric case 
of Slepian-Wolf only. Similar model is also used in [17] for the 
channel with memory. Some work along the lines of the soure 
correlation having Markov structure for the asymmetric case 
of Slepian-Wolf problem has also been done in [18]. For the 
distributed source coding, the results of the previous section 
still hold with R Cl = 1 , R C2 = 1 and different values of 
Cf and Cb- Hence, define (R a , Rb,Cf,Cb)-SW distributed 
code as the code which returns value of a,b,c and R as the 
parameters of the code construction as in previous section 
taking values of R a ,Rb,Cf,Cb by the two step approach in 
equations ([3]) and Q 
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Let us describe the encoding and decoding process in detail 
Encoding: 

We have two sources X\ and X2 each of k bits. The scheme 
has been shown in Figure [8] Send a portion k*alpha of the bits 
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Fig. 8. Encoding scheme for Markov correlation model 



equally spaced directly to the receiver. From the remaining 
k(l-alpha) bits, encode them using {{Rx! -alpha)/(l -alpha), 
(Rx 2 -alpha)/(l -alpha) , Cf, C b )-SW Distributed code defined 
earlier where Rx t , Rx 2 > Cf an d C& are respectively the 
desired source rates for the two sources and the forward and 
backward capacities of the channel between the two sources. 
Decoding: 

The decoding approach has been shown in Figure [9] The 
steps of message parsings in the scheme are: 

1) Message Parsing from the bit nodes to the Markov 
Model: The systematic bits of the two sources and the 
k* alpha bits that were sent without any encoding are 
used to get the likelihoods of the bits of N, as N is the 
sum of the two sources, and hence we can easily get 
their likelihoods from the likelihoods of the bits of the 
two sources. The likelihoods of the nodes of the received 
vector corresponding to the systematic bits of the two 
sources be and b^ respectively with n denoting the 



bit with n varying from 1 to k and the ordering is 
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k(alpha) bits 
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where r\ n is the normalizing factor so that t° + i« = 
1. Now, this message ijj foes back to the systematic 
portion of the bit nodes. Using t® we update the less 
reliable of the systematic bit nodes of the two sources 
by the xor operation. The systematic bits of two sources 
are updated by messages h„ and j* respectively. Let us 
consider message update of h^, j% will be similar. The 
update equations for are given by 
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Fig. 9. Decoding scheme for Markov correlation model 



according to the ordering in the input source vector of 
the two sources, then the likelihoods of the nodes of N 
are given by 



x bl 



x6H 



2) 



Message Parsing from Markov Model to the bit nodes: 
After receiving the noisy version of N, and knowing 
the memory structure of N, we can use trellis de- 
coding to update the likelihoods of N and send the 
likelihoods towards the bit nodes. Consider the trellis 
for a finite-state binary Markov model. The starting 
and ending states associated with a particular edge V 
are represented by s (e) and s E (e), respectively, and 
the bit corresponding to 'e' is denoted by e(e). The 
trellis has two parallel branches between states (one 
associated with the bit e(e) = and the other with the 
bit e(e) = 1). Each one of the branches in the trellis 
will have an associated a priori probability a e , which is 
obtained from the parameters of the Markov Model. The 
resulting equations that implement the belief propagation 
algorithm over the Markov model are given by 

ak-i[s S \e)}a e g^ e) 



otk{s) 



E 

5 E (e) = 



e:s s (e)— s 

*£ = *?n £ a n -i[s s (e)]l3 n [s E (e)}a e gl 

e:e(e)—x 



Since one of a n or b n will be known perfectly (by the 
encoding scheme), this just means to update the other if 
that is not known perfectly. 

3) Message Parsing from the bit nodes to the check nodes: 
This message propagation is from the k(l -alpha) sys- 
tematic bits and the parity bits to the check nodes using 
their respective likelihoods, likelihoods of the systematic 
nodes given by and j% and those of parity bits the 
same as the initial likelihoods in the first iteration and 
those updated from the previous iteration from check 
nodes to bit nodes later. This message parsing is the 
same as the standard belief propagation scheme 

4) Message Parsing from the check nodes to the bit nodes: 
Using the belief propagation, we send the updated 
likelihoods towards the bit nodes. 

Hence, the overall scheme can be summarized as: 

1) Message parsing from bit nodes to Markov model and 
back to the bit nodes. 

2) Message parsing from bit nodes to check nodes. 

3) If the parity equations are not satisfied, and iterations 
are less then maximum-iterations(say 100) go to Step 1 
after message parsing from the check nodes to the bit 
nodes. 
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Fig. 10. Simulation Results 
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V. Simulations of Distributed Source coding for 
Markov correlation 

Take the source correlation model as a Markov source with 
two states Si with probability of as one, and 52 with 
probability of as zero, and the transition probability from 
Si to S2 and from S2 and Si are the same and equal to p, the 
parameter which decides the entropy rate in the plot. Also, 
take the first source to be i.i.d, making Ct = Cb = l-h(p). 
Take k=6250, alpha=.2, and Rx x = Rx 2 = 0.75. Hence, we 
send 6250*. 2=1250 bits of each source without encoding and 
for the remaining bits we use the equations Q and Q with 
R a = R b = .6875, and C f = C b = .5, we get a=2, b=l, c=l, 
R=.8421. The simulation with varying value of p is shown in 
Figure [10] 

VI. Conclusions 

This work deals with the duality between source and 
channel coding. We take ideas developed for channel coding 
and transform them appropriately to construct joint source 
channel coding techniques. Distributed source coding scheme 
is a special case of the above work with the capacity of the 
channels to be equal to 1. We illustrated how to do joint 
distributed source channel coding at arbitrary source rates in 
the Slepian-Wolf rate region with arbitrary memoryless source 
correlation and arbitrary channel. We also illustrated how to do 
distributed source coding at arbitrary point in the Slepian-Wolf 
rate region when the source correlation model has memory. In 
all the simulations, we considered regular (3,x) LDPC codes. 
Taking irregular codes instead may perform better due to 
irregularity in the coding scheme. 

Further work can be done to extend the distributed source 
coding for sources with Markov correlation to joint source 
channel coding. Joint source channel coding for sources with 
memory has been studied in [27] [28]. Some work on the 
asymmetric case for this problem has also been done using 
turbo codes in [29]. But the problem using LDPC codes and 
doing for general source rates are still open problems. 
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